Peer Comparison or Guideline-Based Feedback and Postsurgery Opioid Prescriptions

Key Points Question Can email feedback on opioid prescribing informed by social norms reduce guideline-discordant postoperative opioid prescribing? Findings In this 3-arm cluster-randomized clinical trial that included 640 surgeons, 2 forms of social norm-based email feedback sent to surgeons with postoperative opioid prescriptions above guideline-recommended amounts significantly reduced guideline-discordant prescribing. Meaning Email feedback based on social norms is an effective and light-touch intervention for increasing guideline-concordant opioid prescribing after surgery.


STATEMENT OF COMPLIANCE
The trial will be carried out in accordance with International Conference on Harmonisation Good Clinical Practice (ICH GCP) and the following: United States (US) Code of Federal Regulations (CFR) applicable to clinical studies ( The protocol, informed consent form(s), recruitment materials, and all participant materials will be submitted to the Institutional Review Board (IRB) for review and approval.Approval of both the protocol and the consent form must be obtained before any participant is enrolled.Any amendment to the protocol will require review and approval by the IRB before the changes are implemented to the study.In addition, all changes to the consent form will be IRB-approved; a determination will be made regarding whether a new consent needs to be obtained from participants who provided consent, using a previously approved consent form.

Title:
Postoperative Nudges to Reduce Opioid Prescribing (POST-OP) Study Description: This study tests the effectiveness of two email-based behavioral nudges, one based on peer behavior and one based on best practice guidelines, in reducing excessive opioid prescriptions after surgery.It will be conducted in three surgical specialties (general surgery, orthopedic surgery, and obstetric/gynecological surgery) at 19 hospitals within one healthcare system.At each hospital, these specialties will each be randomized to a control group or one of two nudge groups.Both types of email-based nudges are expected to reduce opioid prescribing after surgery.

Objectives:
Primary Objective: Health Services Research Secondary Objectives: Basic Science Endpoints: Primary Endpoint: Share of discharge opioid prescriptions above prescribing guidelines Secondary Endpoints:

Description of Study Intervention:
Each month for one year, surgeons in the nudge groups will receive emails comparing their opioid prescribing either to their peers' prescribing or to prescribing guidelines.Study Duration: 24  Nudges will be delivered on the third Tuesday of each month for 12 months, October 2021 through September 2022.Data will be collected continuously, with data from the first through last day of the previous calendar month informing whether a nudge is triggered and, if so, the content of the nudge.
After 12 months, the nudges will stop, but data will be collected continuously for another 12 months to assess the persistence of any effects of the nudges.

INTRODUCTION
2.1 STUDY RATIONALE Physician opioid prescribing practices are a major upstream contributor to increases in chronic opioid use, opioid addiction and overdose deaths.Opioid prescribing after surgery is an important intervention target, as opioids are commonly overprescribed after surgery, post-operative opioid prescribing is a risk factor for chronic opioid use, and unused opioids prescribed for patients after surgery are an important reservoir of opioids available for misuse and diversion.

BACKGROUND
2][3][4][5][6][7] This post-operative overprescribing puts both patients and communities at risk, increasing the patient's likelihood of developing chronic opioid use [8][9][10][11][12][13][14] or opioid-induced ventilatory impairmenINT1 and adding to the reservoir of unused opioids available for misuse and diversion. 11,15  discrepancy between clinicians' awareness of the opioid epidemic and the degree of overprescribing-over half of opioid pills prescribed after surgery go unused 7 -suggests that prescribing practices are not based on purely rational decisions.7][18][19] Even well-informed clinicians make cognitive errors when estimating the benefits and harms of treatment, and these errors are especially likely where there is uncertainty about risks and benefits (as with opioid prescribing decisions for individual patients). 20 recent years, behavioral economists and experimental psychologists have successfully leveraged behavioral insights to design "choice architecture" that "nudges" individuals to make better decisions without infringing on their freedom of choice. 21,22 uch behavioral nudges are promising strategies for changing clinician prescribing behavior because they are often more cost-effective than traditional interventions, 23 can be integrated into existing clinical workflows, and are rapidly scalable once built.
One powerful type of behavioral nudge relies on the strong motivation that most people have to conform with their peers' behavior. 24,25 bundant research has found that people (including clinicians) are strongly motivated to adhere to prevailing social norms, 24,25 and that nudges based on describing social norms can be used to influence prescribing decisions. 26other type of behavioral nudge relies on motivation to follow injunctive norms-to do what is considered the "right thing to do".For example, clinicians may be motivated to follow best practice guidelines published by a well-respected organization.Previous studies suggest that such guidelines are in reality often ignored and thus ineffective in changing behavior, 27,28 but there is insufficient evidence to determine whether they are more or less effective than nudges that describe peer behavior.
0][31][32][33][34][35][36][37][38] The results have been promising, but because most of these studies have used a pre-post design, it is possible that the observed decreases in prescribing can be explained by a secular trend.Furthermore, all of these studies have bundled and tested different interventions together (e.g., grand rounds presentations or patient education in addition to nudges), making the effectiveness of the nudges alone unclear.Accordingly, the evidence base for the effectiveness of behavioral nudges in influencing post-operative opioid prescribing is limited.

KNOWN POTENTIAL RISKS
We believe that the risk presented by our nudge interventions is negligible, both because the nudges do not prevent the clinicians from using their own clinical judgment and because previous studies have found that reducing the amount of opioids prescribed after surgical operations did not affect patient satisfaction 39,40 , pain scores [39][40][41] , or refill rates [42][43][44] .

KNOWN POTENTIAL BENEFITS
By simultaneously addressing both opioid overprescribing and the science of provider behavior change, we both improve health and advance science.0][31][32][33][34][35][36][37][38] From a scientific perspective, examining the comparative effectiveness of ways to operationalize and implement norms, and to examine how this varies across provider types and settings, is at the cutting edge of advances in the science of applied behavioral health economics.Research addressing both issues has the potential for high public health and scientific impact.

ASSESSMENT OF POTENTIAL RISKS AND BENEFITS
We believe that our study's significant potential benefits, especially in the current context of the opioid epidemic, far outweigh the minor and unlikely risks it may present to patients.We will conduct a three-arm cluster randomized controlled trial of two behavioral nudges compared to usual post-surgical care.One nudge will provide feedback on the surgeon's prescribing behavior relative to institutional prescribing guidelines (an injunctive norm); the other will provide feedback on their prescribing behavior relative to their peers (a descriptive norm).Three surgical specialties (general surgery, orthopedic surgery, and obstetric/gynecological surgery) within 19 hospitals will be randomized such that all surgeons within a given specialty at a given hospital will receive one of three conditions: control, guideline-based nudge, or peer-based nudge.

SCIENTIFIC RATIONALE FOR STUDY DESIGN
We chose a superiority design in order to make the strongest possible conclusions about whether our nudges are effective and support a policy recommendation as to whether healthcare systems should invest resources in implementing similar nudges.
In our control arm, surgeons will not receive any nudges and will not be informed of the study.By not informing them of the study, we will prevent a Hawthorne effect and obtain an accurate representation of status quo prescribing behavior against which to test the effects of the nudges.

JUSTIFICATION FOR DOSE
Using prescribing data from recent years at Sutter Health, we simulated how many surgeons would receive nudges and how many nudges they would receive under different study design parameters.From this we determined that delivering the nudges monthly achieved a balance between exposing them to the intervention and burdening them with emails.We chose to deliver the nudges by email because this method presents a low burden to surgeons, has been successful in a similar study conducted at Sutter Health, 45 and would be easily scalable beyond Sutter Health if our nudges prove effective.

END OF STUDY DEFINITION
The study is considered to be completed once data has been collected for one year after the date on which the final nudges were delivered.

INCLUSION CRITERIA
Eligible discharge prescriptions meet all of the following criteria: • the patient is at least 18 years old at the date of surgery • the patient is discharged to their home • the surgical procedure has an applicable post-operative opioid prescribing guideline • the surgical procedure is the only surgical procedure performed during the patient's hospital stay • the prescription is for an opioid taken orally (tablets, capsules, or liquid solution)

EXCLUSION CRITERIA
To avoid contamination between the intervention arms, surgeons who operate across multiple surgical specialties (defined as surgeons who performed less than 90% of their total procedures in one specialty between 1 June 2020 and 31 May 2021) will not be eligible.

LIFESTYLE CONSIDERATIONS Not applicable.
5.4 SCREEN FAILURES Not applicable.

STRATEGIES FOR RECRUITMENT AND RETENTION
We will enroll surgeons by screening all surgeons present in electronic health record data from 1 June 2020 through 31 May 2021, and a waiver of informed consent has been granted by the RAND IRB, so surgeon-level recruitment is unnecessary.Hospitals have been recruited and will be retained through communication with hospital-and system-level administrators.

STUDY INTERVENTION DESCRIPTION
Surgeons randomized to our study intervention will receive one of two types of behavioral nudges delivered as monthly emails.The two nudges will be active for twelve months (October 2021-October 2022).
To ensure that the nudges target only inappropriate opioid prescribing, surgeons will receive nudges only when they write opioid prescriptions that exceed post-operative opioid prescribing guidelines developed by multidisciplinary teams at the Mayo Clinic. 32,46,47 (nd personal communication with Professor Elizabeth Habermann, Ph.D., MPH, on opioid prescribing guidelines for caesarean section, March 12, 2021; unreferenced) These guidelines recommend ranges of 5mg oxycodone tablet quantities specific to the procedure performed and are partly based on patient surveys of actual post-operative opioid use.
In both nudge conditions, eligibility for receiving a monthly nudge is contingent upon at least two of the surgeon's patients being discharged with a post-operative opioid prescription exceeding the quantities specified by the Mayo Clinic guidelines.Though it may seem counterintuitive for the descriptive norm nudge to be based implicitly on prescribing guidelines, this choice ensures patient safety and avoids confounding the content of the nudge with the threshold for receiving a nudge.

Intervention arm 1: nudge based on descriptive norms
Surgeons randomized to this condition will receive an email with the following content at the end of each month in which at least two of their patients are discharged with a post-operative opioid prescription that exceeds the prescribing guideline for the procedure performed.
[Subject line: Your peers vs. your opioid prescribing safety record] Dear Dr. [Name], In an effort to reduce opioid use among our surgical patients, Sutter Health is reviewing opioid prescriptions and prescribing patterns for surgeons and will be communicating the findings.
In [month], at least XX of your patients were discharged with opioid prescriptions exceeding the amount prescribed by YY% of your peers for these procedures.
YY% of [specialty] surgeons at Sutter Health prescribe within the ranges below.
We will continue to send you opioid prescribing safety reports.
Sincerely, [Signature(s) of chief medical executive, chief of staff, and/or surgical department chair at the surgeon's hospital] [Table including each procedure type performed by this surgeon in the reference month and the corresponding "Amount prescribed by your peers (5mg oxycodone tablets)", with a footnote stating the conversion factors for hydrocodone and tramadol] The ranges of 5mg oxycodone tablets displayed in the email will be the same as the ranges stipulated by the prescribing guidelines, but this nudge will not include any language about guidelines.

Intervention arm 2: nudge based on injunctive norms
This condition will be identical to the first condition, except the content of the monthly emails will refer to safety guidelines rather than the surgeon's peers.
[Subject line: Best practice guidelines vs. your opioid prescribing safety record] Dear Dr. [Name], In an effort to reduce opioid use among our surgical patients, Sutter Health is reviewing opioid prescriptions and prescribing patterns for surgeons and will be communicating the findings.
In [month], at least XX of your patients were discharged with opioid prescriptions exceeding the amounts recommended by safety guidelines for these procedures.
For patient safety, Sutter Health recommends prescribing within the ranges below for these procedures.Doing so will also meet best practice safety guidelines for post-operative opioid prescribing.
We will continue to send you opioid prescribing safety reports.
Sincerely, [Signature(s) of chief medical executive, chief of staff, and/or surgical department chair at the surgeon's hospital] [Table including each procedure type performed by this surgeon in the reference month and the corresponding "Amount recommended by Sutter Health (5mg oxycodone tablets)", with a footnote stating the conversion factors for hydrocodone and tramadol]

Control arm
Surgeons randomized to the control arm will not receive any nudges and will not be informed of the study.By not informing them of the study, we will prevent a Hawthorne effect and obtain an accurate representation of status quo prescribing behavior against which to test the effects of the nudges.

DOSING AND ADMINISTRATION
Nudges will be delivered monthly through the email software MyEmma.

PREPARATION
The monthly nudges will be prepared by a project coordinator at Sutter Health using data extracted from Sutter Health's electronic health record by a data analyst at Sutter Health.They will be prepared by filling in an email template designed and approved by all study investigators.Code used to produce the list of nudge recipients in each month and the information used to fill in the nudge email template (number of discharges with prescriptions above guidelines, procedures for which the receiving surgeon overprescribed, and percent of surgeons in the receiving surgeon's specialty who were compliant with guidelines) has undergone a quality assurance process in which two investigators independently developed code and compared results.The email software MyEmma will be used to prepare and send the nudges.

MEASURES TO MINIMIZE BIAS: RANDOMIZATION AND BLINDING
The study design has four levels: patients, surgeons, surgical specialties, and hospitals.Randomization will take place at the level of the surgical specialty, using a blocked scheme to ensure that each arm has a balance of large and small hospitals and a sample size of surgeons similar to the other two arms.
All investigators except the Sutter Health data analyst extracting the electronic health record data and the Sutter Health project coordinator preparing the nudges will be blinded to intervention assignment.

STUDY INTERVENTION COMPLIANCE
A project coordinator at Sutter Health will prepare and send the monthly nudges according to the predetermined schedule (sent on the third Tuesday each month from October 2021 through September 2022).This study team member will report to the site principal investigator, Xiaowei Sherry Yan, and the study principal investigator, Katherine Watkins, after each nudge is sent.
6.5 CONCOMITANT THERAPY Not applicable.

DISCONTINUATION OF STUDY INTERVENTION
The study's independent Data Safety Monitoring Board (DSMB) may make a recommendation to discontinue the study intervention at any time based on an assessment of the benefit/risk ratio of study participation.If the study intervention is discontinued, data will continue to be collected for another 12 months from the date of discontinuation.

PARTICIPANT DISCONTINUATION/WITHDRAWAL FROM THE STUDY
Participants may discontinue the study intervention at any time by contacting the investigators (an email address will be provided in each nudge) or by clicking the "Unsubscribe" button at the bottom of any nudge email.Their discontinuation and any reasons given will be recorded by the investigators.Data on their prescribing behavior will still be passively collected through the electronic health record until the study's data collection period ends.

LOST TO FOLLOW-UP
Not applicable.

STUDY ASSESSMENTS AND PROCEDURES
8.1 EFFICACY ASSESSMENTS All efficacy assessments will be conducted using data collected from the Sutter Health electronic health record.Data collected will include: • Unique study IDs of the patient, hospital location, procedural case record, hospitalization, opioid order, performing provider, discharging provider, and prescribing provider All safety assessments will be conducted using data collected from the Sutter Health electronic health record.Data collected will include: • Dates and associated diagnoses of emergency department visits within 30 days of discharge • Discharge opioid prescription refills The study statistician will regularly create three reports for each outcome.Each report will feature a set of ratios indicating the relative frequency of adverse events in a different context.The reports will contain the numerator, denominator, and ratio, each individually reported.Reports will be produced at the end of each month of the trial, including through the 1-year post-intervention period.They will be presented and discussed at half-yearly DSMB meetings.
• Report 1: Adverse events per arm a) Numerator: number of adverse events among intervention arm 1 (INT1) patients Denominator: total number of INT1 patients meeting the inclusion criteria b) Numerator: number of adverse events among intervention arm 2 (INT2) patients Denominator: total number of INT2 patients meeting the inclusion criteria c) Numerator: number of adverse events among the control patients Denominator: total number of control patients meeting the inclusion criteria • Report 2: Adverse events per arm among patients of surgeons who ever received a treatment email a) Numerator: number of adverse events among patients whose surgeon ever received an INT1 email Denominator: total number of patients whose surgeon ever received an INT1 email b) Numerator: number of adverse events among patients whose surgeon ever received an INT2 email Denominator: total number of patients whose surgeon ever received an INT2 email Notes: Ever received an INT1/INT2 email means that the surgeon is included if they received an INT1/INT2 email in any month of the trial.We will also compare the report 2 ratios to the report 1 control ratio (item c above) for additional context.
• Report 3: Adverse events per arm among patients who were prescribed an opioid at discharge and whose surgeon who ever received a treatment email a) Numerator: number of adverse events among patients prescribed opioids at discharge and whose surgeon ever received an INT1 email Denominator: total number of patients prescribed opioids at discharge and whose surgeon ever received an INT1 email b) Numerator: number of adverse events among patients prescribed opioids at discharge and whose surgeon ever received a nINT2 email Denominator: total number of patients prescribed opioids at discharge and whose surgeon ever received an INT2 email c) Numerator: number of adverse events among control patients prescribed opioids at discharge Denominator: total number of control patients prescribed opioids at discharge Notes: Ever received an INT1/INT2 email means that the surgeon is included if they received an INT1/INT2 email in any month of the trial.

DEFINITION OF ADVERSE EVENTS (AE)
Adverse event means any untoward medical occurrence associated with the use of an intervention in humans, whether or not considered intervention-related (21 CFR 312.32 (a)).]

DEFINITION OF SERIOUS ADVERSE EVENTS (SAE)
An adverse event (AE) or suspected adverse reaction is considered "serious" if, in the view of either the investigator or sponsor, it results in any of the following outcomes: death, a life-threatening adverse event, or a persistent or significant incapacity or substantial disruption of the ability to conduct normal life functions.Important medical events that may not result in death, be life-threatening, or require hospitalization may be considered serious when, based upon appropriate medical judgment, they may jeopardize the participant and may require medical or surgical intervention to prevent one of the outcomes listed in this definition.

SEVERITY OF EVENT
For adverse events (AEs) not included in the protocol defined grading system, the following guidelines will be used to describe severity.
• Mild -Events require minimal or no treatment and do not interfere with the participant's daily activities.
• Moderate -Events result in a low level of inconvenience or concern with the therapeutic measures.Moderate events may cause some interference with functioning.• Severe -Events interrupt a participant's usual daily activity and may require systemic drug therapy or other treatment.Severe events are usually potentially life-threatening or incapacitating.Of note, the term "severe" does not necessarily equate to "serious".

RELATIONSHIP TO STUDY INTERVENTION
All adverse events (AEs) must have their relationship to study intervention assessed by the medical professionals on the study team based on temporal relationship and their clinical judgment.The degree of certainty about causality will be graded using the categories below.
• Related -The AE is known to occur with the study intervention, there is a reasonable possibility that the study intervention caused the AE, or there is a temporal relationship between the study intervention and event.Reasonable possibility means that there is evidence to suggest a causal relationship between the study intervention and the AE.• Not Related -There is not a reasonable possibility that the administration of the study intervention caused the event, there is no temporal relationship between the study intervention and event onset, or an alternate etiology has been established.

EXPECTEDNESS
The medical professionals on the study team will be responsible for determining whether an adverse event (AE) is expected or unexpected.An AE will be considered unexpected if the nature, severity, or frequency of the event is not consistent with the risk information previously described for the study intervention.

TIME PERIOD AND FREQUENCY FOR EVENT ASSESSMENT AND FOLLOW-UP
The occurrence of an adverse event (AE) or serious adverse event (SAE) may come to the attention of study personnel during the preparation of monthly adverse event reports or half-yearly presentations to the DSMB.
All AEs will be recorded.Information to be collected includes event description, time of onset, clinician's assessment of severity, and relationship to study.All AEs will be followed to adequate resolution.

ADVERSE EVENT REPORTING
All adverse events will be reported to the DSMB at the subsequent DSMB meeting.

SERIOUS ADVERSE EVENT REPORTING
The principal investigator will immediately report to the sponsor and DSMB any serious adverse event, whether or not considered study intervention related, and must include an assessment of whether there is a reasonable possibility that the study intervention caused the event.
8.4.3 REPORTING UNANTICIPATED PROBLEMS TO PARTICIPANTS Not applicable.

STATISTICAL HYPOTHESES
The null hypotheses are that (1)  != 0, (2)  " = 0, and (3)  !=  " in the hierarchical linear model where  is the patient treated by surgeon  in specialty  at hospital ℎ, 1 %& and 2 %& are indicator variables for whether specialty  at hospital ℎ were assigned to intervention arm one or two respectively, X is a set of patient covariates, Z is a set of surgeon covariates, U is a set of specialty covariates, and W is a set of hospital covariates.
The alternative hypotheses are that (1)  !≠ 0, (2)  " ≠ 0, and (3)  !≠  " .9.2 SAMPLE SIZE DETERMINATION Statistical power to identify effects of the nudges was examined using recent past data from the participating hospitals.We estimated design parameters required by the PowerUpR package in R software, 48 which provides the capability to estimate statistical power for randomized block clustered designs.Examining medication dose, input parameters for the calculation included unconditional intracluster correlations (ICC) for the hospital (ICC=0.005),service line (ICC=0.039),and provider (ICC=0.337)levels; the number of service line groups (up to three per hospital); the number of providers by service line expected to participate in the study; and number of patients per service line.The ICCs were empirically determined from our preliminary data.We assumed that covariates informative of the dosage would explain between 25 percent and 50 percent of the dosage variation at each of the patient, provider, and service line levels (i.e., R 2 between 0.25 and 0.50).We derived statistical power, assuming one third of the service line groups within hospital will be randomly assigned to each study arm (two treatment and one control).We computed power for pairwise comparison of each of the two nudge arms versus the no nudge arm and adjusted our alpha level to account for multiple comparisons (alpha=0.05/2).We will have 80% power to detect significant differences between the intervention conditions of at least a minimum detectable effect size (MDES) = 0.347 standard deviations (SDs) when R 2 =0.25, while R 2 =0.5 would yield a MDES of 0.305.

POPULATIONS FOR ANALYSES
Our main analyses will be conducted on an intention-to-treat dataset including all randomized surgeons.Sensitivity analyses may be conducted on a dataset excluding surgeons who received no nudges.9.4 STATISTICAL ANALYSES 9.4.1 GENERAL APPROACH The effects of each nudge intervention relative to the control group and to each other will be tested using a four-level hierarchical model adjusted for multiple hypothesis testing.

ANALYSIS OF THE PRIMARY AND SECONDARY EFFICACY ENDPOINTS
Our primary outcome is the share of discharge prescriptions that were above the guideline for the respective procedure (see above for how guidelines were identified).Prescribing above guidelines is the outcome to which both nudges are linked (even though the descriptive norm nudge does not explicitly refer to guidelines) and thus a key measure of whether clinician behavior responds to the nudges.We define a prescription as being above guidelines if the morphine milligram equivalent (MME) quantity of opioids prescribed is above the ceiling for the procedure-specific guideline (guidelines range from zero to a ceiling).If no opioid is prescribed at discharge, we will code this as within guideline.
We will also analyze the following secondary outcomes to further understand the effects of the intervention.
• MMEs prescribed at discharge • Days' supply of opioids prescribed at discharge • Share of discharges where any opioid was prescribed • Share of patients on opioids for greater than three months post-discharge • Number of 30-day all-cause emergency department visits • Number of 30-day all-cause hospitalizations • Share of discharge opioid prescriptions above prescribing guidelines in the 12 months after the nudges end

Primary analysis
We will analyze outcomes at the level of the discharge using a four-level hierarchical linear model (HLM), [42] thus capturing the clustering inherent in the study design and data generating process.We will analyze outcomes at the patient level, and patients are nested within surgeons, who are nested within specialties, which are nested within hospitals.Both primary and secondary outcomes will follow this modeling structure.To improve the precision of our estimates, we will also include a set of observable patient covariates (X), surgeon covariates (Z), specialty covariates (U), and hospital covariates (W).For patient , treated by surgeon , in specialty , at hospital ℎ, we consider the following HLM formulation for continuous outcomes  #$%& : 1 %& and 2 %& are indicator variables for whether specialty , at hospital ℎ were assigned to intervention arm one or two respectively.
The key terms in the equation are  ! and  " , the covariate-adjusted treatment effects of arms 1 and 2 relative to the control arm;  !answers research question 1 and  " answers research question 2. We will use an F-test to compare coefficients  ! and  " to answer research question 3. Thus, the effect of each nudge is estimated relative to receiving no nudges and to the other nudge.
The model allows for the possibility that the treatment effect varies across hospitals, as captured by the random effects ( !& ,  "& ).Unexplained variation in each of the levels is captured by the random effects  #$%& ,  $%& ,  %& , and  & .We will initially model these six random effects as independent but will also investigate whether including a covariance structure across these components is appropriate.The coefficients  !,  " , and  ( , capture the influence of the covariates at the patient, surgeon, and specialty respectively, and covariates will be mean centered as appropriate to aid in model interpretation.Covariates may include but are not limited to the following: Level 1: patient age, patient sex, patient comorbidities, procedure type, length of operating time; Level 2: surgeon sex, year of surgeon's medical degree; Level 3: total volume of procedures within the specialty: Level 4: number of beds, urbanicity, proportion of patients on Medicaid.Given that the covariates will not change the estimate of the treatment effect (in expectation), only reduce unexplained variance, we will choose a final pool of covariates that we find to be predictive the primary outcome.Model estimates of the treatment effects will adjust standard errors for clustering due to the due to clustered assignment of the interventions.
For binary outcomes, we implement a hierarchical generalized linear model by including a logit link for Equation (1).Note that the Level 1 error term  #$%& is also eliminated.The concatenated model for all four levels with a binary outcome then reduces to: In the binary outcome version, the parameters  ! and  " again identify the treatment effects of arms 1 and 2 relative to the control arm, with interpretation of these parameters adjusted relative to the link function implemented.

Longitudinal analysis
In addition to assessing the treatment effect averaged over the entire 12-month period, we will also analyze treatment effects by month to assess how the treatment effect evolves over time.For this analysis, we will interact study month indicators with the treatment assignment indicators.

Persistence analysis
We will conduct a secondary analysis to examine whether nudge effects persist once the nudges are discontinued.The data will include the RCT data analyzed in the model above, but also data collected for one year post-intervention (the "persistence period").The analysis model above will be modified by adding an indicator for the RCT period versus persistence period plus interaction terms for period and each nudge to the model.The statistical significance of these interaction terms will be used to assess whether the treatment effect significantly differs post-RCT.

SAFETY ANALYSES Safety endpoints include:
• Dates and associated diagnoses of emergency department visits within 30 days of discharge • Discharge opioid prescription refills The study statistician will regularly analyze these endpoints as outlined in section 8.2.

BASELINE DESCRIPTIVE STATISTICS
Intervention groups will be compared on the proportion of discharge opioid prescriptions above guidelines, proportion of discharges with any opioid prescription, and average discharge opioid prescription quantity during the baseline period (1 June 2020 through 18 October 2021).

PLANNED INTERIM ANALYSES
Any interim analyses conducted during the intervention period will be solely for the purposes of safety monitoring or planning related studies; the intervention will not be altered unless recommended by the study's Data Safety and Monitoring Board.9.4.6 SUB-GROUP ANALYSES We will test for heterogeneity in the treatment effect along several domains.Specifically, we will add terms interacting characteristics of the surgeon with each intervention arm and conduct an F-test of the interaction terms for each nudge.
• Specialty: We will also conduct analyses to test whether the response to each nudge varies by surgeons' specialty.• Volume of surgeries: We will test for heterogeneity by number of surgeries performed over the 12 month study period.We will only include surgeries for which we have guidelines in this count.• Baseline opioid prescribing: We will categorize surgeons based on the portion of their surgeries in the 12 months prior to the start of the intervention that were above guidelines.We expect that the intervention will have a larger effect for surgeon with a higher share of prescription above guidelines.We requested a waiver of informed consent, which was approved by the RAND IRB.Our rationale was as follows: • The research involves no more than minimal risk to participants.
• The waiver would not adversely affect the rights and welfare of the participants, as the planned interventions are "nudges" and providers will be free to prescribe what they deem appropriate.• The research could not practicably be done without a waiver of consent.We expect that allowing providers the opportunity to opt out of this trial would compromise the results of this trial by introducing selection bias.Data collected from the Sutter Health electronic health record will be checked for quality.Quality checks will include searching for duplicated prescriptions, prescriptions with missing key information, and prescriptions for quantities likely to be erroneous.
The code used to produce the list of nudge recipients in each month and the information used to fill in the nudge email template (number of discharges with prescriptions above guidelines, procedures for which the receiving surgeon overprescribed, and percent of surgeons in the receiving surgeon's specialty who were compliant with guidelines) has undergone a quality assurance process in which two investigators independently developed code and compared results.
10.1.9DATA HANDLING AND RECORD KEEPING 10.1.9.1 DATA COLLECTION AND MANAGEMENT RESPONSIBILITIES The following data will be collected by investigators at Sutter Health and the RAND Corporation: • Interview data.The study team will interview Sutter Health providers about their workflows prior to implementation of the nudges and their experiences of the nudges after implementation.Staff at Sutter Health will facilitate arranging the interviews, so we will not be collecting names and contact information for recruitment.• Electronic health record (EHR) data.This study will analyze data on clinician prescribing behavior, patient demographic data, and patient clinical data extracted from Sutter Health's EHR.Study team members employed by Sutter Health will extract the data from the EHR, remove all elements of PHI except dates and hospital names, and share this limited dataset within the study team by uploading it to a folder within the project's Teamspace extranet site.Study team members may store this data locally on their password-protected institutionally issued computers for the purpose of data analysis, but will destroy all copies of limited datasets no later than the completion of the project (31 May 2024).• Pharmacy claims and dispensing data.Study team members at Sutter Health will obtain pharmacy claims and dispensing data via Surescripts, a third-party healthcare information technology organization, so that the study can track changes in opioid dispensing and refills.
The principal investigator, Katherine Watkins, has overall responsibility for the safeguarding of all data, and she will have primary responsibility for familiarizing all the project staff with this data safeguarding plan and procedures.She will follow up with project staff at periodic intervals to ensure that they are in compliance with these data safeguarding procedures.
The study site principal investigator, Xiaowei Sherry Yan, will be responsible for ensuring that the Sutter Health EHR data is collected, stored, and shared in accordance with the procedures outlined above and approved by the RAND IRB.
10.1.9.2 STUDY RECORDS RETENTION Fully deidentified datasets will be retained indefinitely.Limited datasets will be destroyed by the investigators no later than the completion of the project (31 May 2024).
10.1.10PROTOCOL DEVIATIONS A protocol deviation is any noncompliance with the clinical trial protocol or International Conference on Harmonisation Good Clinical Practice (ICH GCP).The noncompliance may be either on the part of the participant, the investigator, or the study site staff.As a result of deviations, corrective actions are to be developed by the site and implemented promptly.It is the responsibility of the study site principal investigator, Xiaowei Sherry Yan, to use continuous vigilance to identify and report deviations within 5 working days of identification of the protocol deviation, or within 5 working days of the scheduled protocol-required activity.Protocol deviations must be sent to the reviewing Institutional Review Board (IRB) per their policies.The site investigator is responsible for knowing and adhering to the reviewing IRB requirements.
10.1.11PUBLICATION AND DATA SHARING POLICY This study will comply with the NIH Data Sharing Policy and Policy on the Dissemination of NIH-Funded Clinical Trial Information and the Clinical Trials Registration and Results Information Submission rule.As such, this trial has been registered at ClinicalTrials.gov(NCT05070338), and results information from this trial will be submitted to ClinicalTrials.gov.In addition, every attempt will be made to publish results in peer-reviewed journals.Data from this study may be requested from other researchers by contacting the principal investigator, Katherine Watkins (kwatkins@rand.org).10.1.12CONFLICT OF INTEREST POLICY Any actual conflict of interest of persons who have a role in the design, conduct, analysis, publication, or any aspect of this trial will be disclosed and managed.Furthermore, persons who have a perceived conflict of interest will be required to have such conflicts managed in a way that is appropriate to their participation in the design and conduct of this trial.
45 CFR Part 46, 21 CFR Part 50, 21 CFR Part 56, 21 CFR Part 312, and/or 21 CFR Part 812) National Institutes of Health (NIH)-funded investigators and clinical trial site staff who are responsible for the conduct, management, or oversight of NIH-funded clinical trials have completed Human Subjects Protection and ICH GCP Training.
Identify eligible surgeons (N=778) based on data from the prior year Administer nudges in accordance with patient eligiblity criteria and nudge triggers Stop administering nudges and begin collecting persistence data These practices are consistent with ICH GCP: • 4.5 Compliance with Protocol, sections 4.5.1, 4.5.2, and 4.5.3• 5.1 Quality Assurance and Quality Control, section 5.1.1 • 5.20 Noncompliance, sections 5.20.1, and 5.20.2.
• Morphine milligram equivalents (MMEs) prescribed at discharge • Days' supply of opioids prescribed at discharge • Share of discharges where any opioid was prescribed • Share of patients on opioids for greater than 3 months postdischarge • Number of 30-day all-cause emergency department visits • Number of 30-day all-cause hospitalizations • Share of discharge opioid prescriptions above prescribing guidelines in the year after the intervention ends

•
Patient sex, racial identity, ethnic identity, and age • Patient tobacco use and chronic opioid use • Patient diabetes, substance use disorder, mood disorder, and pain disorder diagnoses • Patient BMI and ASA physical status rating • Patient insurance status • Procedure name, classification (Elective, Emergent, Return to OR, Urgent, Trauma), service line, date, start time, and completion time • Hospital admission and discharge date and time • Performing provider title, specialty, gender, medical school graduation year, and board certifications • Discharge provider title, specialty, gender, medical school graduation year, and board certifications • Prescribing provider title, specialty, gender, medical school graduation year, and board certifications • Presence of any opioid prescriptions in the 30 days and 12 months prior to surgery • Name, quantity, strength, route, date, and time of opioids prescribed in the 24 hours prior to discharge • Discharge opioid prescription order date, start date, end date, discontinue date, generic name, brand name, strength, quantity, days' supply, class, and refills • Non-opioid discharge prescription order date, start date, end date, discontinue date, generic name, brand name, strength, and quantity • Dates and associated diagnoses of emergency department and inpatient visits within 30 days of discharge • Discharge opioid prescription refills 8.2 SAFETY AND OTHER ASSESSMENTS 10.1.2STUDYDISCONTINUATIONANDCLOSUREThisstudymaybetemporarilysuspendedorprematurelyterminated if there is sufficient reasonable cause.If the study is prematurely terminated or suspended, the Principal Investigator (PI) will promptly inform the Institutional Review Board (IRB) and sponsor and will provide the reason(s) for the termination or suspension.Circumstances that may warrant termination or suspension include, but are not limited to:• Determination of unexpected, significant, or unacceptable risk to participants• Demonstration of efficacy that would warrant stopping • Insufficient compliance to protocol requirements • Data that are not sufficiently complete and/or evaluable • Determination that the primary endpoint has been met• Determination of futilityThe study may resume if concerns are addressed satisfactorily, as determined by the DSMB and IRB.10.1.3CONFIDENTIALITYANDPRIVACYOnlystudyinvestigatorswill have access to patient and surgeon data, and this access will be governed by a data safeguarding plan approved by the RAND IRB to ensure confidentiality and privacy.10.1.4FUTUREUSEOFSTORED SPECIMENS AND DATAFully deidentified datasets will be retained and provided to other investigators upon reasonable request.Limited datasets will be destroyed by the investigators no later than the completion of the project (31 May 2024).10.1.5KEYROLES AND STUDY GOVERNANCEPrincipal investigator: Katherine Watkins (kwatkins@rand.org)Study site principal investigator: Xiaowei Sherry Yan (sherry.yan@sutterhealth.org)10.1.6SAFETY OVERSIGHT Safety oversight will be under the direction of a Data and Safety Monitoring Board (DSMB) composed of individuals with the appropriate expertise, including surgery and pain management.Members of the DSMB will be independent from the study conduct and free of conflict of interest.The DSMB will meet semiannually to assess safety and efficacy data on each arm of the study.The DMSB will operate under the rules of an approved charter that will be written and reviewed at the organizational meeting of the DSMB.At this time, each data element that the DSMB needs to assess will be clearly defined.